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Abstract 


The  Software  Engineering  Institute  (SEI)  is  developing  the  Measurement  and  Analysis  Infra¬ 
structure  Diagnostic  (MAID)  method.  MAID  is  a  criterion-based  evaluation  method  that  facili¬ 
tates  the  examination  and  evaluation  of  data  quality  issues  linked  to  how  the  organization 
implements  measurement  and  analysis  practices. 

This  document  presents  the  set  of  criteria  used  during  a  MAID  evaluation.  The  criteria  serve 
as  a  checklist  for  an  evaluator  to  use  to  rate  the  quality  of  an  organization’s  measurement 
and  analysis  practices  and  the  quality  of  the  measurement  information  that  results  from  the 
implementation  of  those  practices. 

The  evaluation  itself  is  carried  out  by  a  team  following  the  MAID  method.  The  MAID  method 
is  described  briefly  in  this  document.  A  full  description  of  the  method  will  be  published  in  a 
separate  document  in  2010. 
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Introduction 


Background 

Ensuring  measurement  and  analysis  (M  &  A)  information  quality  is  a  challenge  for  most  organizations. 
When  organizations  are  not  aware  of  their  data  quality  levels,  they  cannot  know  the  full  business  impact 
of  poor  or  unknown  data  quality  or  how  to  address  it. 

The  Software  Engineering  Institute  (SEI)  is  developing  the  Measurement  and  Analysis  Infrastructure  Di¬ 
agnostic  (MAID)  method.  MAID  is  a  criterion-based  evaluation  method  that  allows  organizations  to  eva¬ 
luate  key  characteristics  of  their  measurement  programs.  Can  You  Trust  Your  Data?  Establishing  the 
Need  fora  Measurement  and  Analysis  Infrastructure  Diagnostic  describes  the  concept,  rationale,  and 
objectives  of  MAID  [Kasunic  2008a]. 

In  this  document,  we  present  a  set  of  criteria  that  can  be  used  to  evaluate  (1)  the  four  stages  of  the  M  &  A 
process: 

•  M  &  A  Planning 

•  Data  Collection  and  Storage 

•  Data  Analysis 

•  M  &  A  Reporting 

(2)  data  and  information  quality  issues,  and  (3)  M  &  A  impact. 

The  MAID  criteria  have  been  organized  into  five  sections  as  shown  in  Table  1 . 

Table  1  Major  topics  covered  by  criteria  in  each  section  of  this  document 


Section  Major  Topics 


M  &  A  Planning 

• 

• 

• 

• 

• 

Identifying  M  &  A  resource  needs  and  skill  training 

Identifying  M  &  A  information  needs 

Identifying  and  defining  measures  and  measurement  indicators  that  ad¬ 
dress  needs 

Planning  and  scheduling  M  &  A  activities 

Developing  M  &  A  processes  and  procedures  to  perform  M  &  A  activities 

Data  Collection  and  Storage 

• 

• 

• 

Collecting  M  &  A  data  that  address  information  needs 

Ensuring  completeness  and  accuracy  of  the  recorded  data 

Ensuring  that  data  are  stored  securely 

Data  Analysis 

• 

• 

• 

• 

Preparing  the  data  for  analysis 

Transforming  the  data  into  tables  and  graphs  that  can  be  analyzed 

Selecting  the  appropriate  statistical  analysis  approach  to  analyze  the  data 

Conducting  statistical  analyses  of  the  data 

Table  continues  on  next  page 
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Section 


Major  Activities 

M  &  A  Reporting  •  Understanding  the  M  &  A  expertise  level  of  the  stakeholders  who  will  use 

the  M  &  A  information  to  support  decision  making 

•  Transforming  the  data  analysis  results  into  M  &  A  information  that  best 
facilitates  comprehension  by  the  stakeholders  who  will  use  the  information 
to  support  decision  making 

Criteria  for  All  M  &  A  Process  •  Writing  effective  M  &  A  documentation 
or  Procedure  Documentation 


The  criteria  vary  in  terms  of  the  impact  they  might  have  if  met  (or  not  met).  A  companion  document  that 
further  detaiis  the  MAID  method  wiii  be  pubiished  in  2010.  The  Method  document  wiii  provide  guidance 
on  the  weighting  of  criteria. 


To  deveiop  the  criteria,  the  SEI  has  drawn  upon,  synthesized,  and  organized  weii-estabiished  best  prac¬ 
tice  guidance  that  is  aiready  embodied  in  existing  modeis,  standards,  methods,  and  appiicable  best  prac¬ 
tices  (see  Figure  1). 


PSM 

TSP 

PMBOK 

CMMI 

six  Sigma 

GQM/GQ(I)M 


^  ' 
MAID  ^ 


t 


ISO  15939 


Balanced 

Scorecard 


Best  practices  in 

♦  quantitative  analysis 
»  graphical  display 

•  information  packaging 


Figure  1:  Inputs  Considered  for  Development  of  MAID^ 

The  MAID  criteria  were  deveioped  for  use  by  a  team  during  a  MAID  evaluation.  This  team  can  be  com¬ 
posed  of  (a)  individuals  from  within  an  organization  if  such  expertise  exists;  (b)  external  M  &  A  experts 
who  are  contracted  by  the  organization  to  perform  the  MAID  evaluation,  or  (c)  a  combination  of  a  and  b. 

MAID  criteria  are  used  as  part  of  the  MAID  method,  summarized  in  Table  2.  MAID  criteria  are  used  by  the 
MAID  evaluation  team  during  Phase  2,  Document  Evaluation,  and  Phase  3,  On-Site  Evaluation. 


Acronyms  used  in  the  graphic  are  defined  as  follows:  Goal-Question-Metric  (GQM), Goal-Question-Indicator-Metric  (GQ(I)M), 
Project  Management  Book  of  Knowledge  (PMBQK),  Practical  Software  and  System  Measurement  (PSM),  Team  Software 
Process  (TSP),  Capability  Maturity  Model  Integration  (CMMI),  and  International  Qrganization  for  Standards  (ISQ). 
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Table  2  Summary  of  the  MAID  method 


MAID  Phase 

Description 

Collaborative  Planning 

• 

• 

• 

• 

• 

Establish  roles  and  expectations. 

Determine  MAID  scope. 

Identify  client  organization's  data  producers,  measurement  analysts,  and  M  & 

A  consumers  based  on  scope. 

Inventory  M  &  A  documents  to  be  reviewed. 

Determine  schedule. 

Document  Evaluation 

• 

• 

• 

• 

• 

• 

• 

Review  M  &  A  documents  and  data  files  (MAID  team). 

Conduct  question  and  answer  sessions  by  phone  and  email  (MAID  team  and 
client). 

Provide  additional  documents  upon  request  (client). 

Conduct  detailed  collaborative  planning  for  site  visit. 

Finalize  detailed  daily  schedule  for  site  visit. 

Send  emails  to  all  client  interviewees  explaining  what  to  expect. 

Perform  readiness  review. 

On-Site  Evaluation 

• 

• 

Conduct  kick-off  briefing. 

For  each  focus  area  defined  within  the  scope  conduct  the  following  interviews 
and  examinations: 

-  interview  -  data  consumers 

-  interview  -  data  producers 

-  interview  -  data  manager 

-  examination  -  data  storage  mechanism 

-  examination  -  database  contents  (if  not  reviewed  during  phase  2) 

-  interview  -  measurement  analysts 

-  interview  -  M  &  A  information  designers  and  presenters 

Report  Results 

• 

Analyze  MAID  results. 

• 

Prepare  briefing. 

• 

Deliver  briefing. 

• 

Conduct  lessons  learned. 

Sections  1  through  5  of  this  document  present  the  current  set  of  MAID  evaiuation  criteria.  Readers  are 
encouraged  to  provide  their  own  suggestions  for  and  feedback  about  the  criteria.  In  the  future,  we  plan  to 
review  the  feedback  and  publish  a  revised  set  of  criteria. 

Conventions  Used  in  This  Document 

Beginning  with  Section  1 ,  terms  that  appear  in  italics  are  defined  in  the  glossary.  Further  occurrences  of 
the  term  are  not  italicized.  Paragraphs  that  appear  in  unshaded  boxes  describe  the  intention  of  the  set  of 
criteria  that  appear  underneath  it.  Examples  and  explanations  appear  in  shaded  boxes. 
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1  Measurement  and  Analysis  (M  &  A)  Planning 


Organization  Level 

1 .1  Organizational  policies  exist  that  mandate  the  establishment  of  an  organization-wide  measurement 
program. 

1 .2  Organizational  business  goals  are  defined  and  documented. 

1 .3  Stakeholders  of  the  business  goals  are  explicitly  defined. 

1 .4  Organizational  business  goals  are  expressed  in  measurable  terms  so  progress  toward  achieving  a 
goal  can  be  assessed. 

1 .5  Organizational  business  goals  are  kept  current.  That  is,  they  are  reviewed  on  a  periodic  basis  (at 
least  annually)  to  ensure  that  they  reflect  the  current  viewpoint  of  management. 

1 .6  A  measurement  plan^  is  documented.^ 

The  organization  provides  adequate  infrastructure  to  implement  an  organization-wide  mea¬ 
surement  program. 


1 .7  An  educational  effort  is  planned  to  communicate  or  emphasize  the  goals  and  priorities  of  the 
measurement  program  to  obtain  support  and  buy-in  from  affected  individuals. 

1 .8  The  plan  specifies  the  resources  that  are  to  be  allocated  for 

1 .8.1  staffing  M  &  A  personnel'* 

1 .8.2  software  tools 

1 .8.3  data  storage 

1.8.4  reporting 

1.8.5  M&Atraining 

1 .8.5.1  M  &  A  awareness  training  for  all  staff 

1 .8.5.2  M  &  A  skill  training  and  guidance  on  measurement  concepts,  data  collection, 
analysis,  interpretation,  info  packaging,  and  reporting,  based  on  need 


“Plan”  does  not  necessarily  imply  a  single  plan.  The  “plan”  might  take  form  as  a  hierarchy  of  plans — for  example,  a  manage¬ 
ment  plan  may  authorize  and  provide  guidelines  to  planning  groups  that  then  develop  more  detailed,  operational  plans  and 
procedures  for  the  organization’s  measurement  program. 

An  M  &  A  plan  need  not  be  a  separate  document.  It  may  be  part  of  an  integrated  organization  process/measurement  plan. 

M  &  A  personnel  are  individuals  who  are  assigned  to  M  &  A  tasks  such  as  data  collection,  data  graphing,  quantitative  analysis, 
and  so  forth. 
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The  plan  specifies  that  “organization”  information  needs  are  to  be  addressed. 


1 .9  Measurable  business  goals  are  documented  In  the  plan.^ 

1.10  A  structured  approach  Is  followed  to  develop  performance  measures  and  measurement  indicators^ 

[Park  1 996;  Baslll  1 994;  McGarry  2001 ;  Kaplan  1 992;  Florae  1 999,  p.  26]. 

The  documented  approach 

1 .10.1  leads  to  a  prioritized  list  of  Information  needs  associated  with  each  business  goal  of  the 
organization^  and  the  relevant  goals  of  a  parent  organization  or  other  Important 
stakeholders 

1.10.2  Identifies  the  data  sources  that  provide  measurement  Information  that  address  the 
Information  needs 

1.10.3  defines  measurement  Indicators  that  address  Information  needs 

1 .1 0.4  defines  base  measures  and  derived  measures  that  contribute  to  development  of  each 
measurement  Indicator 

1.10.5  Identifies  the  data  sources  of  base  measures 


The  plan  Is  well-organized,  technically  correct,  and  understandable  by  the  Intended  audience. 


1 .1 1  The  plan  Is  structured  well  and  reader-friendly. 

1.12  The  relationship  of  the  plan  to  other  relevant  documents  Is  explained  (e.g..  Its  relationship  to 
enterprise  process/measurement  plan,  prp/'ecf  level  measurement  plans,  and  Improvement  Initiative 
plans). 

1 .1 3  Source  locations  of  reference  documents  specified  In  plan  documents  are  provided  and  readily 
accessible. 

1.14  Explanations  of  measurement  concepts  within  planning  documents  are  correct. 

1 .1 5  Terms  that  may  not  be  understood  by  readers  of  the  plan  are  defined. 

1 .1 6  Acronyms  used  In  the  plan  documents  are  defined. 


Goals  are  listed  in  the  measurement  plan  so  it  is  clear  they  are  the  basis  for  deriving  information  needs  and  measurement 
indicators  that  address  the  information  needs.  In  some  cases,  the  goals  of  an  organization  result  from  the  need  to  support  the 
goals  of  a  parent  organization  to  which  it  is  subordinate. 

The  objective  of  the  structured  approach  is  to  ensure  that  performance  measures  and  indicators  provide  the  information 
needed  to  assess  how  well  the  organization  is  performing  against  the  business  goals  over  time.  The  citations  represent  exam¬ 
ples  of  documented  structured  approaches,  but  there  are  others  and  an  organization  may  have  developed  their  own  docu¬ 
mented  approach  for  selecting  appropriate  measures  and  indicators. 

The  prioritized  information  needs  may  also  include  the  information  needs  inherited  from  a  parent  organization. 
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Effective  project  management  principles  are  followed  to  plan  the  development  of  measure¬ 
ment  program  products  (e.g.,  M  &  A  procedures,  data-collection  mechanisms,  and  storage 
mechanisms). 

1 .1 7  Responsibility  for  developing  and  maintaining  the  M  &  A  plan  is  specified. 

1 .1 8  Authority  for  the  plan  is  specified  (i.e.,  the  approval  requirements). 

1.19  For  all  planned  M  &  A  activities,  the  plan  provides  the  following: 

1.19.1  activity  description 

1 .19.2  description  of  work  products  produced 

1.19.3  activity  start  and  stop  date 

1.19.4  estimated  effort 

1.19.5  completion  criteria® 

1.19.6  responsible  individuals 

1 .20  The  plan  documents  are  baselined  and  then  kept  under  configuration  control. 

1 .21  The  plan  describes  the  triggers  that  will  lead  to  replanning  and  changes  to  the  plan. 

1 .22  The  plan  is  updated  as  the  measurement  process  is  changed. 

1 .23  The  plan  specifies  an  activity  that  results  in  a  description  that  illustrates  the  source,  destination, 
and  transformation  of  measurement  information  throughout  the  organization. 

M  &  A  terminology  is  defined  to  align  with  standards. 

1.24  M  &  A  terminology  is  defined. 

1 .25  Common  M  &  A  standards  and  terminology  are  used  within  the  organization. 

1 .26  The  organization  adopts  M  &  A  terminology  and  methods  defined  by  standards  organizations  such 
as  the  International  Organization  for  Standardization  (ISO). 


Completion  criteria  are  sometimes  referred  to  as  exit  criteria  or  success  criteria. 
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The  plan  specifies  guidelines  for  the  structure,  content,  format,  and  accessibility  of  M  &  A 
processes  and  procedures. 


1 .27  The  plan  mandates  that 


1.27.1 

structure,  content  (e.g.,  who,  what,  when,  where,  why,  and  how),  and  format  of 

1.27.2 

measurement  procedure  documents  are  defined 

location  of  and  access  mechanism  for  M  &  A  process/procedure  documentation  is 
communicated  to  all  affected  personnel 

Detailed  M  &  A  plans  specify  the  content  that  must  be  included  in  M  &  A  procedures 


1 .28  Data  collection  and  storage  procedures  are  documented  that  describe  how  each  base  measure  is 
defined,  collected,  and  stored.  A  data  collection  and  storage  procedure  includes  the  following: 


1.28.1 

role  responsible  for  maintaining  the  procedure  description 

1.28.2 

role  responsible  for  the  data-collection  activity 

1.28.3 

list  of  stakeholders  who  own  the  information  need 

1.28.4 

entity  (e.g.,  specific  process  or  product)  that  is  being  measured 

1.28.5 

attributes  of  the  entity  that  are  to  be  measured 

1.28.6 

base  measure  name 

1.28.7 

definition  of  the  base  measure  as  specified  by  the  following  characteristics: 

1.28.7.1  measurement  method 

1 .28.7.2  type  of  method  (subjective  or  objective) 

1 .28.7.3  scaie  to  be  used 

1.28.7.4  type  of  scaie 

1 .28.7.5  unit  of  measurement 

1.28.8 

format  of  recorded  data 

1.28.9 

time  and  frequency  of  data  collection 

1.28.10 

software  application  used  to  record  data  (e.g.,  spreadsheet,  database,  or  text  document) 

1.28.11 

error-checking  sub-procedure 

1.28.12 

identification  of  the  repository  for  collected  data 

1.28.13 

data  security  requirements 

1.28.14 

data  access  privileges 
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1 .29  A  procedure  is  documented  that  describes  the  analysis  approach  (i.e.,  the  who,  what,  how,  and 
when)  for  each  derived  measure  and  measurement  indicator.  An  anaiysis  procedure  specifies  the 
foiiowing: 


1.29.1 

stakehoider  or  stakehoiders  who  own  the  information  need  (by  organization  and  roie) 

1.29.2 

anaiyst  (roie  description) 

1.29.3 

entity  (e.g.,  specific  process  or  product)  that  was  measured 

1.29.4 

attributes  of  the  entity  that  were  measured 

1.29.5 

pointer  to  the  data  collection  and  storage  procedure 

1.29.6 

frequency  of  analysis 

1.29.7 

base  measures  and  definitions  or  pointer  to  base  measure  characterizations 

1.29.8 

derived  measure  as  characterized  by 

1.29.8.1  measurement  function 

1.29.8.2  indicator 

1 .29.8.3  analysis  model 

1 .29.8.4  decision  criteria 

1.29.9 

repository  location  for  analyzed  M  &  A  results 

1.29.10 

security  requirements  for  stored  M  &  A  results 

1.29.11 

access  privileges  for  stored  M  &  A  results 

1.29.12 

software  tools  used  to  process  and  analyze  the  data,  including  the  version  number  of 
applications 

1.29.13 

file-naming  conventions 

1.29.14 

version  control  requirements 

1.29.15 

storage  location  of  measurement  information  (e.g.,  base,  derived,  and  analysis  results) 

1.29.16 

storage  guidelines 

1 .30  A  procedure  is  documented  that  describes  the  reporting  requirements  for  M  &  A  information.  An 
M  &  A  reporting  procedure  specifies  the  foiiowing: 


1.30.1 

information  need  addressed  by  the  measurement  indicators 

1.30.2 

audience  (stakeholders) 

1.30.3 

measurement  indicators  to  be  reported 

1.30.4 

role  responsible  for  reporting  the  M  &  A  information 

1.30.5 

frequency  of  reporting 
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1 .30.6  mechanism  (e.g.,  presentation,  web  access,  dashboard,  or  report) 

1 .30.7  report  content  (e.g.,  background,  expianation,  current  status,  what  has  changed, 
interpretive  guidance,  recommendations,  other  insights,  and  vaiidation  check  to  ensure 
that  information  need  is  met) 

1 .30.8  repository  iocation  for  M  &  A  reports 

1 .30.9  security  requirements  for  M  &  A  reports 

1 .30.10  access  priviieges  for  stored  M  &  A  reports 

The  M  &  A  pian  specifies  that  procedures  be  deveioped  that  enabie  stabie  impiementation  of 
M  &  A  over  time. 


1 .31  A  vaiidation  mechanism  is  defined  to  ensure  that  each  measure  or  measurement  indicator  satisfies 
the  information  need  it  is  intended  to  address. 

1 .32  An  independent  quaiity  assurance  function  responsibie  for  auditing  the  execution  of  the 
measurement  pian  is  defined. 

1 .33  Taiioring  guideiines  are  documented  that  describe  what,  when,  and  how  M  &  A 
processes/procedures  are  taiiored. 

1 .34  A  procedure  is  documented  that  describes  how  a  waiver  for  confidentiaiity  and  disciosure 
requirements  (for  M  &  A  information)  can  be  obtained. 

1 .35  A  procedure  is  documented  that  describes  how  version  and  configuration  controi  of  M  &  A  artifacts 
is  maintained. 

Specific  for  projects® 

1 .36  A  project  estimation  process^®  is  deveioped  for  estimating 


1.36.1 

size 

1.36.2 

effort 

1.36.3 

staffing 

1.36.4 

quality 

1.36.5 

schedule 

1.36.6 

cost 

Mature  organizations  will  mandate  that  projects  implement  this  in  a  common  way  to  support  valid  comparison  among  projects 
and  support  benchmarking. 

The  quality  of  the  estimation  process  will  depend  on  the  maturity  of  the  organization/project.  Also,  one  important  consideration 
in  the  use  of  a  particular  estimation  technique  is  that  different  techniques  will  apply  in  different  circumstances  [McConnell  2006, 
p.  77]. 
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1 .37  The  plan  specifies  that  projects  collect  data  that  assesses  project  life-cycle  performance  including 
the  following: 

1.37.1  effort 

1 .37.2  cost 

1 .37.3  productivity 

1.37.4  schedule  predictability 

1 .37.5  requirements  completion  ratio  (i.e.,  functionality  delivered) 

1 .37.6  defect  density  by  phase 

1 .37.7  post-release  defect  density 
[Kasunic  2008b] 

1 .38  The  plan  specifies  that  project  characterization  data  (e.g.,  influence  factors  found  in  the  SEI  report 
A  Data  Specification  for  Software  Project  Performance  Measures:  Resuits  of  a  Coiiaboration  on 
Performance  Measurement  [Kasunic  2008b]  and  adjustment  factors  in  COCOMO  II)  is  to  be 
identified  and  collected  by  projects  to  support  benchmarking  and  continuous  improvement  of 
project  estimation  processes  including: 

1.38.1  artifact  reuse 

1 .38.2  project  type 

1.38.3  application  domain 

1 .38.4  average  team  size  (tracked  throughout  the  project  life  cycle) 

1.38.5  maximum  team  size 

1 .38.6  team  expertise 

1 .38.7  process  maturity 

1 .39  The  plan  specifies  that  if  and  when  a  commercial  software  estimation  tool  or  some  other  parametric 
model  (such  as  COCOMO  II)  is  used,  it  is  calibrated  with  one  of  the  following  approaches 
[McConnell  2006]: 

•  using  historical  data  generated  by  previous  and  similar  projects  within  the  organization 

•  using  industry-based  data  appropriately,  if  historical  data  are  unavailable 

•  using  a  group  of  experts  employing  a  structured  approach  (e.g.,  wide-band  Delphi),  if  data  are 
not  available 

•  using  individual  expert  judgment,  if  neither  data  nor  a  group  of  experts  is  available. 

1 .40  The  plan  specifies  that  actuals  (i.e.,  effort,  cost,  schedule,  and  quality)  be  compared  with  estimates 
and  the  outcomes  documented. 
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1 .41  The  plan  specifies  that  the  following  are  recorded: 

1 .41 .1  number  of  software  requirements  changes  per  time  unit  and/or  per  life  cycle  phase 

1 .41 .2  rationale  for  a  requirements  change^^ 

1 .42  The  plan  specifies  that  peer  review  activities  are  conducted  to  identify,  characterize,  and  record 
defects  throughout  the  project  life  cycle. 

1 .43  The  plan  specifies  that  individuals  who  contribute  to  product  development  collect  data  to 
characterize  their  personal  performance  [Humphrey  1999]. 

1 .44  The  plan  specifies  that  personal  performance  data  are  combined  to  characterize  team  performance 
[Humphrey  1999]. 


"  Doing  so  supports  cause  and  effect  analysis. 

This  refers  to  any  software  product  artifact  including  requirements,  design,  code,  and  documentation. 
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2  Data  Collection  and  Storage  Criteria 


For  Data  That  is  Collected’^ 

2.1  For  each  measure  that  is  collected,  a  data-collection  process/procedure  includes  the  following 
information: 

2.1 .1  role  responsible  for  maintaining  the  procedure  description^'* 

2.1 .2  role  responsible  for  the  data-collection  activity 

2.1 .3  a  list  of  stakeholders  who  own  the  information  need 

2.1 .4  entity  (e.g.,  specific  process  or  specific  product)  that  is  being  measured 

2.1 .5  attributes  of  the  entity  that  are  to  be  measured 

2.1.6  base  measure  name 

2.1 .7  definition  of  the  base  measure  as  specified  by  the  following  characteristics: 

2. 1.7.1  measurement  method 

2.1 .7.2  type  of  method  (subjective  or  objective) 

2.1 .7.3  scale  to  be  used 

2.17.4  type  of  scale 

2. 1.7. 5  unit  of  measurement 

2.1 .8  format  of  recorded  data 

2.1 .9  time  and  frequency  of  data  collection 

2.1 .10  software  application  used  to  record  data  (e.g.,  spreadsheet,  database,  or  text  document) 

2.1 .1 1  error-checking  sub-procedure*^ 

2.1 .12  repository  for  collected  data 

2.1.13  data  security  requirements 

2.1 .14  data  access  privileges 

2.1.15  special  instructions  (e.g.,  mandatory  vs.  optional  recording  of  information) 


These  criteria  do  not  address  data  collection  for  special  cases  such  as  inferential  statistical  studies  which  would  imply  the  need 
for  random  sampling  plans. 

The  author  of  the  procedure  is  identified  in  the  M  &  A  plan.  In  some  cases  the  author  and  maintainer  of  the  procedure  are  the 
same. 

This  ensures  all  data  fields  were  recorded  or  transcribed  correctly. 
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2.1.16  requirement  that  missing  data  designated  as  mandatory  to  coilect  be  accompanied  by 
the  reason  information  was  not  recorded 

2.1.17  fiie-naming  conventions 

2.1 .18  computer  system  where  recorded  data  resides  and  path  name  where  data  are  stored 

2.1 .19  data-collection  form  to  be  used,  with  a  copy  provided  in  process/procedure  document  and 
pointers  to  on-line  version 

2.1 .20  instructions  for  completing  the  form 

2.1 .21  definitions  for  the  data  fields  located  on  the  form,  when  needed 

2.1 .22  instructions  for  submitting  the  completed  data-collection  form 

2.2  A  data-collection  form/template  accompanied  by  clear  instructions  is  used  to  collect  the 
measurement  data.  The  form  contains  the  following  fields: 

2.2.1  a  reference  to  the  data-collection  procedure 

2.2.2  physical  location  where  data  are  collected,  if  applicable 

2.2.3  identity  of  specific  individual  who  collects  data,  with  contact  information 

2.2.4  date  and  time  stamp 

2.2.5  information  identifying  the  organization  and  project 

2.2.6  entity  name 

2.2.7  attribute  name 

2.2.8  location  or  source  of  the  measures 

2.2.9  confidentiality  level  of  collected  data 

2.3  The  data  form  is  consistent  with  the  associated  data-collection  procedure. 

2.4  When  data  are  shared  among  databases,  the  metadata  that  characterizes  the  source  of  the  data 
are  available.  For  example, 

2.4.1  organizational  entity  (enterprise,  organization  name,  or  project  name) 

2.4.2  entity 

2.4.3  attribute 

2.4.4  time  data  was  collected 

2.4.5  identity  of  the  data  collector^® 


One  of  the  basic  principles  of  quality  is  accountability  by  persons  performing  the  data  collection  or  reporting  work.  Even  auto¬ 
mated  integrity  rules  that  might  be  implemented  cannot  be  effective  without  accountability  [English  2004]. 
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2.5  An  audit  of  data-collection  practices  is  periodicaiiy  conducted  to  ensure  that  measurement  and 
anaiysis  procedures  are  being  adhered  to. 

2.6  A  data  quaiity  audit  is  reguiariy  performed  to  quantify  and  correct  data  recording  errors  or  data 
entry  errors. 

2.7  A  checkiist  or  similar  mechanism  exists  to  monitor  compliance  to  the  data-collection  requirements. 

The  checklist  includes  the  following: 

2.7.1  organization  entity  name 

2.7.2  contact  information  for  individual  responsible  for  collecting  the  information  (if  data  are 
being  collected)  or  reporting  the  measurement  information  (if  data  are  being  reported) 

2.7.3  measures  reported 

2.7.4  due  date  (for  reported  data) 

2.7.5  actual  date  (for  reported  data) 

2.7.6  number  of  data-collection  errors  by  severity 

2.7.7  percentage  of  mandatory  fields  completed  with  valid  data^^ 

2.8  The  number  of  data-collection  errors  is  monitored  over  time. 

2.9  Security  mechanisms  exist  to  ensure  that  stored  data  cannot  be  altered  by  unauthorized 
individuals. 


In  this  context,  “valid  data”  is  used  to  mean  data  that  is  free  of  transcription  errors. 
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3  Data  Analysis  Criteria 


3.1  Data  analysis  process  or  procedure  specifies  the  following  for  each  information  need: 

3.1 .1  stakeholders  who  own  the  information  need  (by  organization  and  role) 

3.1 .2  analyst  (role  description) 

3.1 .3  entity  (e.g.,  specific  process  or  product)  that  was  measured 

3.1 .4  attributes  of  the  entity  that  were  measured 

3.1 .5  pointer  to  the  data-collection  and  storage  procedure 

3.1.6  frequency  of  analysis 

3.1 .7  base  measures  and  definitions  or  pointer  to  base  measure  characterizations 

3.1 .8  derived  measure  as  characterized  by 

3. 1.8.1  measurement  function 

3. 1.8. 2  indicator 

3. 1.8. 3  analysis  model 

3. 1.8. 4  decision  criteria 

3.1 .9  repository  location  for  analyzed  M  &  A  results 

3.1 .10  security  requirements  for  stored  M  &  A  results 

3.1 .1 1  access  privileges  for  stored  M  &  A  results 

3.1 .12  software  tools  used  to  process  and  analyze  the  data,  with  version  number 

3.1.13  file-naming  conventions 

3.1.14  version  control  requirements 

3.1.15  storage  location  of  measurement  information  (e.g.,  base,  derived,  or  analysis  results) 

3.1.16  storage  guidelines 
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3.2  When  a  dataset  is  examined 

3.2.1  a  data  value  is  not  missing  from  a  data  cell  unless  entry  has  been  specified  as  optional 

3.2.2  data  format  of  each  data  cell  has  been  correctly  set  or  applied,  if  applicable 

3.2.3  data  values  are  within  the  permissible  range  of  values  for  the  data  cell,  if  applicable 

3.2.4  formula  computations  with  a  data  cell  have  been  correctly  set  up,  if  applicable 

3.3  Before  using  a  statistical  method,  the  dataset  distribution  is  examined  to  expose  data  outliers  (if 
they  exist)  and  to  ensure  that  departures  from  underlying  assumptions  associated  with  the  method 
are  understood  and  documented^®  [NIST  2009a,  NIST  2009b]. 

3.4  When  quantitative  descriptive  statistics  are  used,  the  measure  of  central  tendency  is  accompanied 
by  a  measure  of  variability. 

3.5  In  cases  where  a  special  study^®  is  undertaken,  a  research  plan  is  developed  that  describes  the 
following: 

3.5.1  problem  statement 

3.5.2  method  or  approach  to  solving  the  problem 

3.5.3  analysis  method 

3.5.4  reporting  method 

3.6  Model  adequacy  is  checked  using  residual  analysis  before  assuming  that  the  results  of  ANOVA 
analysis  are  valid. 


Techniques  for  testing  the  departure  from  assumptions  are  available  [NIST  2009b]. 

Special  studies  are  those  that  are  considered  sporadic  analysis  studies  conducted  to  address  specific  problems  or  issues. 
Special  studies  include  those  that  are  carried  out  to  produce  quantitative  projections  of  key  project  attributes  and  evaluate  the 
accuracy  and  realism  of  project  plans.  Also  special  studies  include  those  activities  that  are  chartered  as  a  consequence  of  a 
corrective  action  (resulting  from  routine  data  analysis)  to  fix  a  problem  or  to  explore  an  improvement  opportunity. 
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3.7  When  developing  measurement  instruments  (e.g.,  questionnaires  for  surveys  or  test  forms)  the 
analyst  assesses  the  reliability  of  the  measures  and  indicators. 

Explanation 

There  are  four  general  classes  of  reliability  estimates,  each  of  \A/hich  estimates 
reliability  in  a  different  way. 

Inter-rater  or  inter-observer  reiiabiiity  is  used  to  assess  the  degree  to  which 
different  raters  or  observers  give  consistent  estimates  of  the  same  phenome¬ 
non. 

Test-retest  reiiabiiity  is  used  to  assess  the  consistency  of  a  measure  from  one 
time  to  another. 

Paraiiei-forms  reiiabiiity  is  used  to  assess  the  consistency  of  the  results  of  two 
tests  constructed  in  the  same  way  from  the  same  content  domain. 

Internal  consistency  reliability  is  used  to  assess  the  consistency  of  results 
across  items  within  a  test. 

3.8  When  using  statistical  modeling  methods,  adherence  to  the  following  underlying  assumptions 
required  for  validity  of  the  results  are  tested: 

3.8.1  independence 

3.8.2  normality 

3.8.3  constant  variability  of  the  error  terms 

3.9  When  a  statistical  model  is  developed  (e.g.,  linear  regression),  confidence  intervals  are  calculated 
and  displayed  to  illustrate  the  uncertainty  associated  with  the  fitted  regression  line  (the  average 
dependent  variable  values). 

3.10  When  a  statistical  model  is  used  (e.g.,  linear  regression)  for  prediction,  a  prediction  interval  is 
calculated  and  displayed  to  illustrate  the  uncertainty  associated  with  the  prediction  of  new 
dependent  variable  values. 
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4  Measurement  and  Analysis  Reporting  Criteria 


4.1  An  audience  analysis  is  conducted  to  ensure  that  M  &  A  information  is  designed  for  the  needs, 
interests,  and  backgrounds  of  the  individuals  for  whom  it  is  intended  [Rossi  2001]. 


Explanation:  Each  M  &  A  stakeholder  is  characterized  according  to  audience  type  so  M  &  A  information  is 
packaged  in  a  way  that  takes  into  account  the  expertise  level  of  the  audience. An  audience  analysis  an¬ 
swers  the  questions  listed  below. 

•  What  is  your  audience's  experience  with  the  topic? 

•  What  terms  and  concepts  will  they  probably  not  understand? 

•  What  do  you  need  to  tell  them  so  that  they  understand  your  meaning? 

•  What  misconceptions  might  they  have? 

[Perelman  2009] 


4.2  For  each  information  need  identified,  a  reporting  process/procedure  defines  the  following: 

4.2.1  role  responsible  for  reporting  the  M  &  A  information 

4.2.2  identification  of  each  stakeholder  to  whom  the  information  is  reported  (i.e.,  the  owners  of 
the  information  need) 

4.2.3  the  measures  and/or  indicators  that  will  be  reported  for  each  stakeholder 

4.2.4  frequency  that  M  &  A  information  is  to  be  reported,  determined  by  the  timeliness 
necessary  to  meet  decision-making  objectives,  and  the  allowable  duration  between  data 
collection  and  reporting 

4.2.5  format  for  reporting  measurement  information  to  each  stakeholder 

4.2.6  mechanism  for  reporting  measurement  information  to  stakeholder  (e.g.,  stand-alone 
report,  dashboard,  or  presentation) 

4.2.7  security,  confidentiality  level,  or  access  rights  of  reported  M  &  A  information 

4.3  A  validation  procedure  (i.e.,  feedback  mechanism)  is  defined  to  ensure  that  information  needs  are 

satisfied  by  the  suite  of  M  &  A  indicators. 

4.4  M  &  A  information  that  is  communicated  to  support  decision  making  are  accompanied  by 

explanations  that  clarify  the  message. 


^  See  Appendix  A  for  an  example  framework  for  characterizing  audience  types. 

Whenever  a  table  or  graph  does  not  speak  clearly  enough  on  its  own,  its  design  should  be  improved  or  explanatory  text  should 
be  added  [Few  2004,  p.  126]. 
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4.5 


M  &  A  information  that  is  communicated  to  support  decision  making  is  accompanied  by 
recommendations  for  what  could  or  should  be  done  based  on  that  information^^  [Few  2004,  p.  127]. 


4.6  When  a  measurement  indicator  value  exceeds  the  acceptability  threshold,  corrective  action  is 

taken. 

4.7  The  communicator  of  M  &  A  information  recognizes  and  reports  opportunities  for  further 

exploration,  further  speculation,  and  valuable  questions  that  decision-makers  ought  to  be 

considering  [Few  2004,  p.  127]. 

4.8  The  M  &  A  report  includes 

4.8.1  author  of  the  report  on  each  page  of  the  report  [Few  2004,  pp.  128-129] 

4.8.2  page  number  on  each  page  of  report  [Few  2004,  pp.  128-129] 

4.8.3  type  of  quantitative  and  categorical  information  that  the  report  represents  on  every  page 
[Few  2004,  pp.  128-129] 

4.8.4  date  or  period  when  the  data  was  collected  [ISO  2007;Few  2004,  p.  129] 

4.8.5  limitations  of  the  results  and  any  other  qualifications  (e.g.,  limitations  to  the  validity  of  the 
conclusions  drawn)  [ISO  2007] 

4.8.6  names  and  versions  of  software  tools  used  for  performing  statistical  analysis  [ISO  2007] 

4.8.7  number  of  observations  from  which  conclusions  are  drawn  [ISO  2007] 

4.8.8  sampling  procedures  that  are  used  [ISO  2007] 

4.8.9  assumptions  underlying  the  analysis  techniques  that  are  used  and  the  results  of  any 
sensitivity  analysis  performed  to  check  for  robustness  to  violation  of  assumptions  [ISO 
2007] 

4.8.10  precisely  how  aggregates  are  performed  (e.g.,  average  or  weighted  average)  [ISO  2007] 

4.8.1 1  unit  of  observation  about  which  conclusions  are  drawn  (e.g.,  inspection  package  or 
configuration  item)  [ISO  2007] 

4.8.12  how  missing  data  and  anomalies  were  dealt  with,  where  applicable  [ISO  2007] 

4.8.13  how  outliers  were  dealt  with  during  data  analysis,  where  applicable  [ISO  2007] 

4.8.14  how  combining  data  across  different  datasets  was  performed,  where  applicable  [ISO 
2007] 

4.8.15  for  any  statistical  tests,  whether  they  are  one-  or  two-sided  [ISO  2007] 


In  some  cultures,  recommendations  may  not  be  welcome  from  the  presenter  of  data.  Such  a  consideration  must  be  factored  in 
within  the  context  of  the  situation. 

Some  information  should  be  included  on  every  page  of  a  report  because  excerpts  from  multi-page  reports  are  often  copied  and 
distributed.  If  the  information  that  identifies  the  report’s  contents  only  appears  at  the  beginning,  readers  will  have  no  way  of 
knowing  what  they  are  seeing  when  they  have  only  a  portion  of  the  report  [Few  2004,  p.  128]. 
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4.8.16  for  any  statistical  tests,  the  alpha  levels  used  (i.e.,  the  amount  of  acceptable  error)  [ISO 
2007] 

4.8.17  for  any  statistical  tests,  how  p  values  are  calculated  (i.e.,  the  probability  of  getting  the 
observed  result  or  a  more  extreme  one  by  chance)  [ISO  2007] 

4.8.18  how  confidence  intervals  are  calculated,  where  applicable  [ISO  2007] 

4.8.19  how  prediction  intervals  are  calculated,  where  applicable 

4.9  Text  answering  the  following  questions  should  be  included  on  every  page  of  an  M  &  A  report: 

4.9.1  What?  (Description  of  the  type  of  quantitative  and  categorical  information  that  the  report 
represents.) 

4.9.2  When?  (A  listing  of  the  range  of  dates  the  information  represents;  the  point  in  time  when 
the  information  was  collected.) 

4.9.3  Who?  (A  person  to  contact  if  readers  have  questions.) 

4.9.4  Where? (A  page  number.  Within  the  report  the  best  format  is  page  #  of  ##.) 

[Few  2004,  p.  128] 


Explanation:  M  &  A  information  is  reported  in  one  of  the  three  ways  (a)  in  a  sentence,  (b)  in  a  table,  (c)  in  a  graph. 
The  remaining  criteria  are  organized  into  the  following  categories: 

•  tables 

•  graphs 

•  tables  and  graphs 


Tables 

4.1 0  Tables  are  used  to  present  quantitative  information  when 

•  the  table  is  used  to  look  up  individual  values 

•  the  table  is  used  to  compare  individual  values 

•  precise  values  are  required 

•  quantitative  information  to  be  communicated  involves  more  than  one  unit  of  measure 
[Few  2004,  p.  46,  Robbins  2005,  p.  344] 

4.1 1  Table  titles,  column  headings,  and  footnotes  precisely  define  what  each  data  point  in  the  table 
means  [Klass  2002]. 

4.12  When  rates  or  ratios  are  reported,  both  the  numerator  and  denominator  are  clearly  defined  [Klass 
2002]. 


Excerpts  from  multi-page  reports  are  often  copied  and  distributed.  If  the  information  that  identifies  the  report’s  contents  only 
appears  at  the  beginning,  readers  will  not  have  a  way  of  knowing  what  they’re  seeing  when  they  have  only  a  portion  of  the  re¬ 
port  [Few  2004,  p.  128]. 
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4.1 3  To  delineate  columns  and  rows  in  a  table, 


4.13.1  white  space  alone  is  used  whenever  space  allows 

4.13.2  only  enough  vertical  white  space  between  groups  to  make  breaks  noticeable  is  used 

4.13.3  subtle  fill  colors  are  used  when  white  space  cannot  be  used 

4.13.4  subtle  rules  are  used  when  fill  colors  cannot  be  used 

4.13.5  grids  are  avoided  altogether 

[Few  2004,  pp.  135-136,  139-158;  Klass  2002;  Ford  2007] 

4.14  Arranging  data  in  a  table 

4.14.1  Columns  and  rows 

4.1 4.1 .1  sets  of  categorical  subdivisions  are  arranged  across  separate  columns  if  they 
are  few  in  number  and  the  maximum  number  of  characters  in  those 
subdivisions  is  not  too  large  [Few  2004,  pp.  141-142] 

4.1 4.1 .2  time-series  subdivisions  are  arranged  horizontally  across  separate  columns 
[Few  2004,  pp.  142-143] 

4.14.1 .3  ranked  subdivisions  are  arranged  vertically  down  rows  [Few  2004,  pp.  142- 
143] 

4.14.2  Groups  and  breaks 

4.14.2.1  column  headers  are  repeated  at  beginning  of  each  new  group  [Few  2004,  pp. 
143-144] 

4.14.2.2  each  group  starts  on  a  new  page  when  groups  should  be  examined 
independently  [Few  2004,  pp.  142-143] 

4.14.3  Column  sequence 

4.14.3.1  sets  of  categorical  subdivisions  arranged  down  the  rows  of  a  single  column 
are  placed  to  the  left  of  quantitative  values  associated  with  them  [Few  2004, 
p.  144] 

4.14.3.2  sets  of  categorical  subdivisions  that  have  a  hierarchical  relationship  (e.g., 
between  product  families  and  products)  are  placed  from  left  to  right  to  reflect 
that  hierarchy  [Few  2004,  pp.  144-145] 

4.1 4.3.3  quantitative  values  calculated  from  another  set  of  quantitative  values  are 
placed  just  to  the  right  of  the  column  from  which  they  were  derived  [Few 
2004,  p.  144] 

4.14.3.4  columns  containing  data  that  should  be  compared  are  placed  close  to  each 
other  [Few  2004,  p.  145;  Ford  2007]. 
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4.14.4  Value  sequence 

4.14.4.1  When  categorical  subdivisions  have  a  meaningful  order,  they  are  sorted  in 
that  order  [Few  2004,  p.  146;  Klass  2002] 

4.1 5  Formatting  text  in  a  table 

4.15.1  Orientation 


4.15.1.1 

Text  orientation  other  than  horizontal  (left  to  right)  is  avoided  [Few  2004,  p. 
146] 

4.15.2  Alignment 


4.15.2.1 

numbers  are  aligned  to  the  right,  keeping  decimal  points  aligned  as  well  [Few 
2004,  p.  147] 

4.15.2.2 

calendar  dates  are  aligned  to  the  left  using  a  format  that  maintains  a  constant 
width  [Few  2004,  p.  148] 

4.15.2.3 

text  is  aligned  to  the  left  [Few  2004,  p.  148] 

4.15.2.4 

non-numeric  data  are  centered  if  they  all  have  the  same  number  of 
characters  and  the  number  of  characters  in  the  header  is  significantly  greater 
[Few  2004,  p.  148] 

4.15.3  Number  formatting 

4.1 5.3.1  comma  is  placed  to  the  left  of  every  three  whole  number  digits^^  [Few  2004, 


4.15.3.2 

p.  149] 

whole  numbers  are  truncated  by  sets  of  three  digits  to  the  nearest  thousand, 
million,  billion,  and  so  forth  whenever  numeric  precision  can  be  reduced 
without  the  loss  of  meaningful  information,  and  this  is  declared  so  in  the  title 
or  header  (e.g.,  U.S.  dollars  in  thousands)  [Few  2004,  p.  149] 

4.15.3.3 

negative  sign  or  parentheses  is  used  to  display  negative  numbers  (e.g.,  - 
8,395.37  or  (8,395.37)),  but  if  parentheses  are  used,  the  numbers  that  are 
enclosed  are  right  aligned  with  the  positive  numbers  [Few  2004,  p.  149] 

4.15.3.4 

percentage  sign  is  placed  immediately  to  the  right  of  every  percentage  value 
[Few  2004,  p.  149] 

Note  that  in  some  languages  this  is  not  the  convention.  Instead,  a  period  might  be  used.  There  may  be  other  differences  de¬ 
pending  on  the  culture  or  language. 
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4.15.4  Date  formatting 

4.1 5.4.1  months  are  expressed  either  as  a  two-digit  number  or  a  three-digit  character 
word  [Few  2004,  p.  150] 

4.15.4.2  days  are  expressed  as  two  digits  [Few  2004,  p.  150] 

4.15.5  Number  and  date  precision 

4.1 5.5.1  precision  of  numbers  or  dates  does  not  exceed  the  level  needed  to  serve  the 
communication  objectives  and  the  needs  of  the  readers^®  [Few  2004,  p.  151 ; 
Klass  2002,  Wainer  1997] 

4.15.6  Font 

4.1 5.6.1  legible  font  is  selected^^  [Few  2004  p.  1 52;  Klass  2002] 

4.1 5.6.2  same  font  is  used  throughout  the  table  [Few  2004  p.  1 52] 

4.15.7  Emphasis  and  color 

4.1 5.7.1  boldface,  italics,  or  change  in  color  of  fonts  is  used  to  group  or  highlight  data 
[Few  2004,  p.  153;  Klass  2002] 

4.16  Summarizing  values  in  a  table 

4.16.1  columns  containing  group  summaries  are  made  visually  distinct  from  detail  columns  [Few 
2004,  p.  153] 

4.16.2  summaries  are  placed  in  the  group  header  if  its  rows  extend  down  multiple  pages  [Few 
2004,  p.  154] 

4.1 7  Providing  page  information  in  a  table 

4.17.1  column  headers  are  repeated  at  the  top  of  each  page  [Few  2004,  p.  156] 

4.17.2  current  row  headers  are  repeated  at  the  top  of  each  page  [Few  2004,  pp.  156-157] 

Graphs 

4.1 8  Graphs  are  used  to  present  quantitative  information  when 

•  the  message  is  derived  from  the  shape  of  the  values 

•  the  graph  is  used  to  reveal  relationships  among  multiple  values 
[Few  2004,  p.  46] 


There  are  no  precise  guidelines  regarding  this  since  it  is  based  on  communication  objectives  which  would  need  to  be  learned 
through  interviews. 

Examples  of  fonts  that  have  good  legibility  are  Serif:  Times  New  Roman,  Palatino,  and  Courier;  Sans-Serif:  Arial,  Verdana,  and 
Tahoma.  Examples  of  fonts  that  have  poor  legibility  are  Serif:  Script,  Broadway,  Old  English;  Sans-Serif;  Gill  Sans  Ultra,  Papy¬ 
rus,  Tempus  Sans  ITC  [Few  2004,  p.  153]. 
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4.1 9  The  quantitative  scaie  on  a  graph  starts  at  zero  uniess  there  is  a  need  to  show  smaii  differences 
between  iarge  vaiues.  In  the  iatter  case,  the  reader  is  aierted  that  the  graph  does  not  provide  a 
contextuaiiy  accurate  visuai  representation  of  the  vaiues  so  that  readers  can  adjust  their 
interpretations  [Few  2004,  p.  169]. 

4.20  Three-dimensionai  graphics  are  not  used  to  dispiay  quantitative  information^®  [Coies  1997;  Few 
2004,  p.  170;  Robbins  2005,  pp.  19-27;  Rossi  2001]. 

4.21  The  number  of  categoricai  subdivisions  in  a  graph  is  iimited  to  between  five  and  eight  [Few  2004, 
p.  207]. 

4.22  When  other  text  is  used  in  a  graph  (e.g.,  tities  or  notes)  the  text  appears  as  ciose  to  the  information 
that  it  complements  as  possible  without  interfering  with  that  information  [Few  2004,  p.  206]. 

4.23  Graphical  captions  are  comprehensive  and  informative.  A  framework  for  a  caption  that  can 
contribute  to  such  a  clear  explanation  is  as  follows: 

•  describe  everything  that  is  graphed 

•  draw  attention  to  important  features  of  the  data 

•  describe  the  conclusions  that  are  drawn  from  the  data  on  the  graph 

[Cleveland  1994,  p  55] 

Explanation:  Graphs  vary  primarily  in  terms  of  the  types  of  relationships  they  are  used  to  communicate.  Essential¬ 
ly,  there  are  seven  types  of  relationships  that  graphs  are  typically  used  to  display.  These  types  are  described  in  the 
table  below  [Few,  pp.  65-66]. 


This  graph  type ... 

Expresses ... 

Nominal  comparison 

the  comparative  sizes  of  multiple  related  but  discrete  values  in  no  particular 
order 

Time  series 

the  rise  and  fall  of  values  through  time 

Part-to-whole 

the  portion  of  each  part  relative  to  the  whole 

Deviation 

how  and  the  degree  to  which  one  or  more  things  differ  from  another 

Distribution 

a  range  of  values  as  well  as  the  shape  of  the  distribution  across  a  range 

Correlation 

how  two  paired  sets  of  values  vary  in  relation  to  one  another 

4.24  For  nominal  comparisons,  bar  graphs  or  data  points  are  used  [Few  1994,  p.  70]. 


This  refers  to  business  graphics  such  as  bar,  column,  line,  and  pie  charts.  However,  in  some  cases,  three-dimensional  graphs 
are  appropriate,  such  as  graphs  generated  as  a  result  of  response  surface  methodology. 
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4.25  For  time  series  designs 

4.25.1  lines  or  points  and  lines  are  used  to  emphasize  overall  patterns 

4.25.2  vertical  bars  are  used  to  emphasize  and  compare  the  values  associated  with  an 
individual  point  in  time  rather  than  with  the  overall  pattern  of  values  as  they  change 
through  time 

4.25.3  points  connected  by  lines  are  used  to  slightly  emphasize  individual  values  while  still 
highlighting  the  overall  pattern 

4.25.4  time  is  always  placed  on  the  horizontal  axis 

[Few  1994,  pp.70-72] 

4.26  For  ranking  designs,  bars  (either  horizontal  or  vertical)  are  only  used  [Few  1994,  pp. 72-76]. 

4.27  For  part-to-whoie  designs 

4.27.1  bars  (either  horizontal  or  vertical)  are  used 

4.27.2  pie  charts  are  not  used  to  impart  quantitative  information. 

[Cleveland  1994,  pp.  262-263;  Few  2004,  p.  60;  Few  2006,  pp.  134-135;  Tufte  2001 

p.  178;  Becker  1996;  Robbins  2005,  pp.  3-7;  ProcessTrends  2008;  Bertin  1983] 

4.27.3  stacked  bar  charts  are  not  used  to  impart  quantitative  information  unless  there  is  a  need 
to  display  multiple  instances  of  a  whole  and  its  parts,  with  emphasis  primarily  on  the 
whole  [Cleveland  1994,  pp.  265-267;  Robbins  2005,  pp.  29-33;  Few  2004, p.  74;  Few 
2006,  pp.  135-137]. 

4.28  For  deviation  designs 

4.28.1  lines  are  used  to  emphasize  the  overall  pattern  only  when  displaying  deviation  and  time 
series  relationships  together 

4.28.2  points  connected  by  lines  are  used  to  slightly  emphasize  individual  data  points  while  also 
highlighting  the  overall  pattern  when  displaying  deviation  and  time-series  relationships 
together 

4.28.3  bars  are  used  to  emphasize  individual  values,  but  are  limited  to  vertical  bars  when  a  time 
series  relationship  is  included 

4.28.4  a  reference  line  is  always  included  to  compare  the  measures  of  deviation  against 

[Few  1994,  pp.76-77] 

4.29  For  distribution  designs:  singie  distributions 

4.29.1  vertical  bars  are  used  to  emphasize  individual  values 

4.29.2  lines  are  used  to  emphasize  the  overall  shape 

[Few  1994,  pp.78-80] 
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4.30  For  distribution  designs:  muitipie  distributions 

4.30.1  vertical  or  horizontal  bars  are  used  (i.e.,  range  bars  or  boxes)  to  encode  the  full  range 
from  the  low  value  to  the  high  value,  or  some  meaningful  portion  of  the  range  (e.g.,  90% 
of  the  values)^® 

4.30.2  points  or  lines  are  used  together  to  encode  measures  of  the  center  (e.g.,  the  median) 
[Few  1994,  pp.80-82] 

4.31  For  correiation  designs,  points  and  a  trend  line  are  used  in  the  form  of  a  scatter  plot  [Few  1994, 
pp. 83-84]. 

4.32  Important  quantitative  information  is  highlighted  in  graphs  using  one  of  the  techniques  presented  in 
the  table  below  [Few  2004,  p.  120]. 


Attribute 

Values  Useful  for  Emphasis 

Line  width 

Thicker  lines  (including  words  and  numbers  that  are  boldfaced)  stand 
out  more  than  thinner  lines. 

Orientation 

Slanted  words  and  numbers  (i.e.,  italics)  stand  out  more  than  those  that 
are  oriented  normally  (i.e.,  not  slanted),  assuming  that  vertically 
oriented  type  is  the  norm. 

Size 

Bigger  objects,  words,  and  numbers  stand  out  more  than  smaller  ob¬ 
jects. 

Enclosure 

Objects,  words,  and  numbers  that  are  enclosed  by  lines  or  background 
fill  color  stand  out  more  than  those  that  are  not  enclosed. 

Hue 

Objects,  words,  and  numbers  that  have  any  hue  that  are  different  from 
the  norm  stand  out. 

Color  intensity 

Objects,  words,  and  numbers  that  are  bright  stand  out  more  than  those 
that  are  light  or  pale. 

4.33  When  the  orientations  of  line  segments  on  a  graph  are  judged  to  decode  information  about  rate  of 
change,  the  line  segments  are  banked  to  45  degrees  [Cleveland  1994,  p.  70]. 

4.34  Error  bars  on  graphs  are  clearly  explained^”  [Cleveland  1 994,  p  59]. 

4.35  A  logarithmic  scale  is  used  when  it  is  important  to  understand  percent  change  or  multiplicative 
factors  [Cleveland  1 994  pp.  95-1 04;  Frees  1 997;  Frees  1 996,  Ch.  6;  Few  2004,  pp.  1 97-202]. 


An  example  is  the  box  and  whisker  plot  (i.e.,  box  plot,  boxplot,  and  box  and  whisker  diagram)  [Tukey  1977,  p.  139]. 

Error  bars  can  convey  one  of  several  possibilities:  (1 )  The  sample  standard  deviation  of  the  data;  (2)  An  estimate  of  the  stan¬ 
dard  deviation  (also  called  the  standard  erroi)  of  a  statistical  quantity;  (3)  A  confidence  inten/ai  for  a  statistical  quantity  [Cleve¬ 
land  1994,  p  59]. 
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Tables  and  Graphs 

4.36  Whenever  quantitative  information  is  presented  in  tabies  or  graphs,  the  data  itseif  stands  out 
clearly,  without  distraction.  This  involves  eliminating  anything  that  doesn’t  represent  the  data, 
except  for  visual  devices  that  support  the  data  in  a  necessary  way.  In  that  case,  they  should  be 
displayed  in  muted  fashion  so  as  to  not  distract  from  the  data  itself  [Tufte  2001,  Few  2004,  pp.  117- 
120,  Cleveland  1994,  pp.  25-54]. 

4.37  When  descriptive  statistics  are  reported,  a  measure  of  variation  accompanies  the  measure  of 
central  tendency  [Few  2004,  p.  25]. 

4.38  The  reported  M  &  A  information  emphasizes  how  the  attributes  of  interest  have  changed  since  the 
previous  reporting  events®^  [Tufte  1997,  p.  30;  Tufte  2001 ,  pp.  74-75]. 


Preferably,  the  measurement  information  will  illustrate  the  trend  of  the  data  since  the  last  reporting  event,  showing  what  has 
changed  over  the  last  several  reporting  periods  (i.e.,  it  will  show  trends). 
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5  Criteria  for  Aii  M  &  A  Process  or  Procedure  Documentation 


5.1  Individuals  who  are  responsible  for  writing  M  &  A  process  and  procedures  have  received  training  in 
developing  technical  documentation. 

5.2  M  &  A  process  and  procedure  descriptions  are  appropriate  to  users’  needs.  As  evidence,  one  or 
more  of  following  activities  is  conducted  as  part  of  the  process  or  procedure  development  process: 

•  audience  and  task  analysis 

•  technical  reviews 

•  user  questionnaires 

•  observation 

•  usability  testing®^ 

[Techscribe  2006,  Perelman  2009,  Horn  1982] 

5.3  M  &  A  process/procedure  descriptions  contain  easily  accessible  information  [Horn  1 982, 
Techscribe  2006]. 

5.4  M  &  A  process/procedure  information  is  chunked  into  small  manageable  units^®  [Horn  1982]. 

5.5  M  &  A  process/procedure  information  is  clearly  labeled  for  quick  retrieval  [Horn  1982]. 

5.6  Cross-referencing  to  other  sections^'^  is  avoided  in  process/procedure  documentation^®  [Horn 
1982]. 

5.7  Glossaries,  style  guides,  templates,  and  controlled  language  are  used  to  make  M  &  A  process  and 
procedure  descriptions  linguistically  accurate  and  stylistically  consistent  [Techscribe  2006]. 


This  is  sometimes  referred  to  as  pilot  testing. 

“  A  manageable  unit  of  information  is  one  consisting  of  no  more  than  nine  pieces  of  information.  Research  suggests  that  people 
can  best  process  and  remember  no  more  than  seven  plus  or  minus  two  (7  ±  2)  pieces,  or  units,  of  information  at  one  time. 

That  is,  the  reader  should  not  be  forced  to  reference  other  sections  of  the  document  or  scroll  to  a  different  part  of  the  docu¬ 
ment.  However,  cross-referencing  to  information  via  hyperlinks  is  fine. 

Forcing  readers  to  other  parts  of  the  same  document  and  other  documents  to  perform  the  task  that  is  the  purpose  of  the  pro¬ 
vided  information  should  be  avoided  [Horn  1984]. 
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Request  for  Feedback 


The  SEI  is  interested  in  feedback  from  and  collaboration  with  organizations  that  would  like  to  pilot  the 
MAID  method.  If  you  would  like  to  provide  suggestions,  feedback,  or  discuss  collaboration,  please  contact 
SEI  info@sei.cmu.edu. 
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Appendix  A:  Example  of  Audience  Type  Classification 


Audience  Type 

Description 

Experts 

People  who  understand  M  &  A  theory  and  practice.  Often,  they  have  advanced  de¬ 
grees  or  training  in  measurement  and  analysis. 

Specialists 

People  who  possess  limited  training  on  how  to  maintain  and  interpret  a  specified 
number  of  M  &  A  indicators. 

Non-specialists 

People  who  have  the  least  technical  knowledge.  Their  interests  are  entirely  practical 
and  they  want  to  understand  M  &  A  theory  only  enough  to  use  it  to  support  decision 
making. 
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Glossary 


Analysis  model 

An  algorithm  or  calculation  involving  two  or  more  based  or  derived 
measures  with  associated  decision  criteria.  Analysis  models  produce 
estimates  or  evaluations  relevant  to  defined  information  needs 
[McGarry  2001,  p.  23]. 

Base  measure 

A  measure  defined  in  terms  of  an  attribute  and  the  method  for  quantify¬ 
ing  it.  A  base  measure  is  functionally  independent  of  other  measures 
[ISO  2007]. 

Decision  criteria 

Thresholds,  targets,  or  patterns  used  to  determine  the  need  for  action 
or  further  investigation  or  to  describe  the  level  of  confidence  in  a  given 
result  [ISO  2007]. 

Derived  measure 

A  measure  defined  as  a  function  of  two  or  more  base  measures  [ISO 
2007]. 

Indicator 

A  measure  that  provides  an  estimate  or  evaluation  of  specified 
attributes  derived  from  a  model  with  respect  to  defined  information 
needs  [ISO  2007]. 

Measurement  function 

An  algorithm  or  calculation  performed  to  combine  two  or  more  base 
measures  [ISO  2007]. 

Measurement  method 

A  logical  sequence  of  operations,  described  generically,  used  in  quanti¬ 
fying  an  attribute  with  respect  to  a  specified  scale.  The  type  of  mea¬ 
surement  method  depends  on  the  nature  of  the  operations  used  to 
quantify  an  attribute.  Two  types  may  be  distinguished:  (a)  subjective:  a 
quantification  involving  human  judgment,  and  (b)  objective:  a  quantifica¬ 
tion  based  on  numerical  rules  [ISO  2007]. 

Measurement  procedure 

A  set  of  operations,  described  specifically,  used  in  the  performance  of  a 
particular  measurement  according  to  a  given  method  [ISO  2007]. 

Objective  (measurement  me¬ 
thod) 

A  quantification  based  on  numerical  rules  such  as  counting.  These  rules 
may  be  implemented  by  human  or  automated  means.  For  example, 
lines  of  code  may  be  quantified  by  counting  semi-colons  [McOarry 

2001,  p.  21]. 

Organization 

An  administrative  structure  in  which  people  collectively  manage  one  or 
more  projects  as  a  whole,  and  whose  projects  share  a  senior  manager 
and  operate  under  the  same  policies  [CMMI  2006]. 

37  I  CMU/SEI-2009-TR-022 


Project 

A  managed  set  of  interrelated  resources  that  delivers  one  or  more 
products  to  a  customer  or  end  user  [CMMI  2006].  A  temporary  endea¬ 
vor  undertaken  to  create  a  unique  product,  service,  or  result  [PMI  2008]. 

Scale 

An  ordered  set  of  values,  continuous  or  discrete,  or  a  set  of  categories 
to  which  an  attribute  is  mapped.  The  type  of  scale  depends  on  the  na¬ 
ture  of  the  relationship  between  values  on  the  scale.  Four  types  of 
scales  are  commonly  defined:  (a)  nominal  -  the  measurement  values 
are  categorical;  (b)  ordinal  -  the  measurement  values  are  rankings;  (c) 
interval  -  the  measurement  values  have  equal  distances  corresponding 
to  equal  quantities  of  the  attribute;  (d)  ratio  -  the  measurement  values 
have  equal  distances  corresponding  to  equal  quantities  of  the  attribute 
where  the  value  of  zero  corresponds  to  none  of  the  attribute  [ISO  2007]. 

Subjective  (measurement 
method) 

A  quantification  involving  human  judgment  or  rating.  For  example,  rely¬ 
ing  on  an  expert  to  rate  the  complexity  of  functions  as  high,  medium  or 
low  is  a  subjective  method  of  measurement  [McGarry  2001 ,  p.  21]. 

Type  of  scale 

Four  types  of  scales  are  commonly  defined:  (a)  ratio,  (b)  interval,  (c) 
ordinal,  (d)  nominal  [ISO  2007;  Roberts  1979]. 

Unit  of  measurement 

A  particular  quantity,  defined  and  adopted  by  convention,  with  which 
other  quantities  of  the  same  kind  are  compared  in  order  to  express  their 
magnitude  relative  to  that  quantity  [ISO  2007]. 
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